Skip to content

feat(nzbdav): two-phase migration — import + symlink rewrite#522

Merged
javi11 merged 8 commits intomainfrom
feat/nzbdav-two-phase-migration
Apr 22, 2026
Merged

feat(nzbdav): two-phase migration — import + symlink rewrite#522
javi11 merged 8 commits intomainfrom
feat/nzbdav-two-phase-migration

Conversation

@javi11
Copy link
Copy Markdown
Owner

@javi11 javi11 commented Apr 20, 2026

Summary

  • Phase 1 — resilient import: New import_migrations table tracks every nzbdav import by GUID, surviving queue resets. Scanner uses it for dedup (replaces FilterExistingNzbdavIds), sets SkipArrNotification=true, and strips nzbdav_id from queue metadata. Service marks rows imported/failed on queue completion.
  • Phase 2 — symlink rewrite: POST /import/nzbdav/migrate-symlinks walks an arr library, detects symlinks targeting <nzbdav_mount>/.ids/<guid>, looks up the AltMount final_path in import_migrations, and atomically rewrites them. BackfillFromImportQueue seeds the table from historical queue rows so already-migrated users can jump directly to Phase 2.
  • Cleanup: id_linker.go deleted, HandleIDMetadataLinks call removed, all .ids/ write paths removed from metadata/service.go (read path preserved for backward compat), FilterExistingNzbdavIds dropped from queue repo.
  • Frontend: Phase 2 card with dry-run preview and Apply button appears in the NZBDav import section after Phase 1 completes or when migration_stats.imported > 0.

Test plan

  • go test -race ./internal/... passes
  • bun run check && bun run build passes
  • Upload a nzbdav sqlite DB via UI → confirm import_migrations rows flip pending → imported, no .ids/ symlinks created, arr not notified
  • Phase 2 dry run with a library containing nzbdav symlinks → verify planned rewrites shown in UI
  • Phase 2 apply → readlink library files confirm new targets point to AltMount mount path
  • On a DB with existing import_queue.metadata.nzbdav_id rows but no import_migrations, hit Phase 2 endpoint → backfill kicks in and symlinks get rewritten
  • GET /import/nzbdav/status includes migration_stats field

javi11 added 8 commits April 20, 2026 10:25
Adds a generic import_migrations table to track two-phase migration
state (Phase 1 = import NZBs into AltMount; Phase 2 = rewrite arr
library symlinks). Includes goose migrations for SQLite and PostgreSQL,
ImportMigration model + status consts, a full ImportMigrationRepository
with Upsert/MarkImported/MarkFailed/MarkSymlinksMigrated/LookupByExternalID/
ListByStatus/Stats/ExistsForSource/BackfillFromImportQueue, and wires
MigrationRepo into the DB struct.
…avIds

Delete HandleIDMetadataLinks and id_linker.go; remove UpdateIDSymlink and
RemoveIDSymlink methods from MetadataService; remove the .id sidecar write
block from WriteFileMetadata (read path preserved for Phase 2 compatibility);
remove FilterExistingNzbdavIds from QueueRepository, Repository, and the
BatchQueueAdder interface; remove related call sites in nzbfilesystem MOVE
handler and nzbdav scanner processBatch.
- Add MigrationRecorder interface to scanner package with UpsertMigration
  and IsMigrationCompleted methods
- Update NzbDavImporter to accept MigrationRecorder as second constructor
  parameter alongside BatchQueueAdder
- processBatch now checks IsMigrationCompleted before enqueueing (skip
  already-imported/symlinks_migrated items) and calls UpsertMigration for
  new items, then strips nzbdav_id from the queue item metadata leaving
  only extracted_files if present
- createNzbFileAndPrepareItem sets SkipArrNotification=true on all items
- batchQueueAdapterForImporter gains migrationRepo field and implements
  MigrationRecorder via ImportMigrationRepository.Upsert/LookupByExternalID
…lure

Wire s.database.MigrationRepo into handleProcessingSuccess and
handleProcessingFailure so that import_migrations rows are marked
imported/failed when the corresponding queue item finishes. Both
calls are non-fatal: failures are logged as warnings and do not
affect the main processing outcome. If no matching migration row
exists (non-nzbdav import), the UPDATE simply affects 0 rows.
Adds RewriteLibrarySymlinks in internal/importer/migration to walk a
library directory and atomically rewrite arr symlinks that point at
<nzbdav_mount>/.ids/<guid> to the final altmount path.  Introduces
DBSymlinkLookup adapter in internal/database, a new
POST /import/nzbdav/migrate-symlinks handler, and enriches the existing
status endpoint with migration_stats when available.
Phase 1: import_migrations DB table tracks every nzbdav import by GUID,
surviving queue resets. Scanner uses it for dedup (replaces
FilterExistingNzbdavIds), sets SkipArrNotification=true, and strips
nzbdav_id from queue metadata. Service marks rows imported/failed on
queue item completion.

Phase 2: POST /import/nzbdav/migrate-symlinks walks an arr library,
detects symlinks targeting <nzbdav_mount>/.ids/<guid>, looks up the
AltMount final_path in import_migrations, and atomically rewrites them.
BackfillFromImportQueue seeds the table from historical queue rows so
already-migrated users can skip directly to Phase 2.

Cleanup: id_linker.go deleted, HandleIDMetadataLinks call removed, all
.ids/ write paths removed from metadata service (read path preserved for
backward compat), FilterExistingNzbdavIds dropped.

Frontend: Phase 2 card with dry-run preview and Apply button appears
after Phase 1 completes or when migration_stats.imported > 0.
Users who already migrated via the old nzbdav flow can now access
the symlink rewrite tool directly from a top-level 'Migrate Symlinks'
tab without needing to run Phase 1 first.
… add Clear All Migrations

- Phase 2 symlink rewrite inside the NZBDav import flow is now gated on
  an empty queue (total_queued + total_processing == 0) with a live
  "Waiting for queue to finish" banner so users don't rewrite symlinks
  while imports are still in flight.
- Split the NZBDav section into nested tabs (Import / Migrate Symlinks).
  The standalone tab runs against already-imported rows for users who
  imported previously and only need to rewrite the arr library.
- Extract the symlink form into a shared SymlinkMigrationForm component
  reused by both entry points.
- Add DELETE /api/import/nzbdav/migrations + UI button to wipe all
  nzbdav migration rows, for the case where the user deleted imported
  files from AltMount and wants to force a full re-import.
@javi11 javi11 merged commit 39116c8 into main Apr 22, 2026
2 checks passed
@javi11 javi11 deleted the feat/nzbdav-two-phase-migration branch April 22, 2026 10:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant